Predictive impairment monitor system and method

ABSTRACT

A driver monitor system and method for predicting impairment of a user of a vehicle. The system includes video cameras, an input device for inputting a list of medications being taken by the driver. Processing circuitry predicts side effects of the medications based on the half-life of the medication, detecting eye gaze movement, eye lid position, and facial expression of the user using images from the video camera, predicting whether the user is transitioning into an impaired physical state that is a side effect of the medications, verifying the side effect of the medications, determining whether the user is fit to drive using the verified side effects of the medications, and outputting to the vehicle an instruction to operate the vehicle in a level of automation that makes up for the at least one side effect or to perform a safe pull over operation of the vehicle.

BACKGROUND Technical Field

The present disclosure is directed to a software application and vehiclesystem that utilizes information of medications being taken by thedriver in order to predict whether or not the driver can operate thevehicle, and can monitor conditions of the driver to determine whetherthe conditions may be side effects of a medication. The vehicle systemmay take corrective action when the conditions of the driver are sideeffects of a medication.

Description of the Related Art

The “background” description provided herein is for the purpose ofgenerally presenting the context of the disclosure. Work of thepresently named inventors, to the extent it is described in thisbackground section, as well as aspects of the description which may nototherwise qualify as prior art at the time of filing, are neitherexpressly or impliedly admitted as prior art against the presentinvention.

Driving under the influence of prescription drugs, some over-the-counter(OTC) drugs, and some herbal products, may potentially be unsafe due toside effects such as drowsiness, slowed reaction times, and coordinationdifficulties. For certain types of prescription drugs, over-the-counterdrugs, and even certain herbel products. driving may be just asdangerous as driving under the influence of alcohol. Similar to alcohol,certain types of drugs and herbel products can directly affect adriver's ability to drive safely.

Drivers may be unaware of the side effects of medications they aretaking and typically believe that side effects will only occur when theyare taken in excess. Drivers may be unable to accurately self-assesstheir impairment when taking medication.

Many prescription medications that are used to control the symptoms ofanxiety and depression (Antidepressants and Antianxiety Agents), such asselective serotonin reuptake inhibitors (SSRI), affect the brain and/orjudgment. Prescription medications such as Valium and Xanax may have atranquilizing effect that can impair judgment and reaction times. Forexample, taking 10 mg of diazepam (Valium) can cause impairment similarto having a blood alcohol content of 0.10%, which is above the legallimit for driving. Antihistamines have a potential driving impairmentdue to a threat of anticholinergic cognitive issues. Although OTCmedications like Claritin, Allegra, and Zyrtec are now manufactured asnon-drowsy, not all antihistamine medications are non-drowsy. Patientsmay not be aware of the effects of the particular antihistamine they aretaking and may just assume that most antihistamines are non-drowsy.Also, antihistamines can cause blurred vision because they dry up thetear ducts. Antihypertensives (Blood pressure medications) may have sideeffects such as lightheadedness, dizziness, and fatigue that can hinderdriving performance. Also, beta-blockers and sympatholytic drugs likeclonidine, guanfacine, and methyldopa may cause sedation, confusion, orinsomnia. Most antipsychotic agents may impair driving performancethrough various central nervous system (CNS) effects. In particular,antipsychotics generally have 3 side effects including sedatiion,anticholinergic effects, and extrapyramidal effects that can directlyaffect a patient's ability to drive. Benzodiazepines (anti-anxietydrugs) can lead to driving impairments in vision, attention, and motorcoordination. Long-acting benzodiazepines can impair psychomotorfunction the following day.

It is a common misconception that stimulants such as caffeine pills andhigh caffeine energy drinks such as Red Bull would be good to takebefore driving. In reality, such stimulants makes a person moreimpetuous and less likely to pay attention to fine details (i.e., lowerability to concentrate). Also, stimulants combined with alcohol can givea person the worst of both. Despite not feeling as drunk , the alcoholis still impairing a person's ability to drive.

Also, nearly 20 percent of seniors commonly take medications that canimpair driving. Medications that are often taken by seniors includebenzodiazepines, narcotic pain medications, hypnotics(anti-depressants), and sleep medications. In addition, seniors mayunintentionally take multiple medications that have the same effect,amplifying the results to unsafe levels. Some seniors may see a numberof physicians and specialists, so they may not be aware of thepotentially unsafe dosage levels of multiple medications.

Driver fatigue or drowsiness is a state of a driver characterized bylowering of the eyelids (typically measured by percent of eyeclosure—PERCLOS), possibly in combination with other signs such as lowerheart rate, yawning, and head dropping. Sometimes a driver sufferingfrom driver fatigue or drowsiness may shake his/her head in an effort tofight off fatigue. In some cases, when the vehicle is stopped, such asat a stop light, the driver may close his/hers eyes for a short periodin an effort to rest their eyes.

Development is underway in more advanced vehicle safety features with anend goal of achieving autonomous vehicles, or self-driving cars. A rangeof lowest to highest levels of automation have been defined. The levelsof autonomous vehicles may use various external vehicle sensors (e.g.,cameras, LiDAR, radar, ultrasonic, GPS, etc.) to scan the externalenvironment. FIG. 1 is a diagram of the range of levels of automateddriving. A lowest level of automation (Level 1) relates to features ofautomatic cruise control 111, advanced emergency braking 113, laneassist 115, and cross traffic alert 117. This level makes use of camerasfor surround view object detection 119 to assist in parking a vehicle. Anext level (Level 2) includes a greater degree of shared control betweenautomation and the driver of the vehicle, such as where the drivercontrols steering, but the vehicle has control over engine power andbraking to maintain and vary speed. This next level includes a functionof traffic jam assist 121 and automatic parking 123. At this next level,automatic parking may include automatic parallel parking, automatic backin parking, or front end parking in a parking spot. Another level ofautomation (Level 3) includes a condition where the vehicle is underfull control but the driver monitors the driving and can intervene atany time, also referred to as conditional automation. This levelincludes advanced emergency braking in combination with steering 125,highway autopilot 131, and remote parking 133. A higher level ofautomation (Level 4) may include a city driving autopilot function 141and a valet parking function 143. At this higher level of automation thedriver may turn their attention to other tasks while the vehicle isperforming driving control. A highest level of automation (Level 5), viaa fully autonomous vehicle, is contemplated in which no humanintervention is required to drive the vehicle. The vehicle operates inauto pilot 151 in which it is expected to work on all roads in allweather conditions. Vehicle control systems have been developed that canswitch from low levels to greater levels of autonomous driving based onthe state of the driver.

U.S. Pat. No. 10,065,658 relates to bias of physical controllers in asystem. The system determines the state of a user from user input thatincludes a calendar of the user, a profile of the user, and/or analysisof audible input from the user. The profile of the user can include alist of medications taken. A controlled action, such as engagingautopilot, may be taken when a threshold condition on the state of theuser has been met. In one case, a user may tell a device that he or shehas taken, e.g., an antihistamine, and thus a physical controller mayvibrate with greater intensity to suggest a possible deviation fromexpected behavior based on this information. A physical controller maychange its sensitivity to resist motions that are closer to a dangerzone of use.

U.S. Patent Application Publication 2019/0202464 relates to vehiclesystems and methods for detecting and mitigating an incapacitateddriver. The disclosed vehicle system determines a specific cause of adriver's incapacitation and operates a vehicle in an at least partiallyautonomous driving mode based on the specific cause of incapacity. Thesystem determines the cause of the driver's incapacitation based on thedriver's behavior as opposed to specific data entries made by thedriver.

Methods have been proposed for detecting driver fatigue and applyingstimulus to mitigate the fatigued state. U.S. Pat. No. 9,302,584 relatesto drowsy driver prevention systems and methods. The disclosed systemdetermines a level of driver drowsiness by monitoring driver behavior,for example, by angle of orientation of the driver's head. Driverprofiles may be created that contain a list of medications and suchfactors may be used to establish thresholds for indicating levels ofconfidence of driver drowsiness.

Although driver fatigue or drowsiness is a condition which may render adriver not fully fit for driving, medications may have other sideeffects that may also impair a driver's behavior. As mentioned above,some medications may have an effect in which vision becomes blurry.Other side effects may include dizziness, confusion, reduced attention,and reduced motor skills.

Thus, there is a need for a system and method of predicting driverbehavior based on medication that the driver has taken and the half-lifeof that medication.

An aspect is an apparatus including an input device by which a driver ofa vehicle inputs medication being taken by the driver, and a processorthat predicts ability of the driver to safely operate the vehicle basedon the half-life of the medication and prevents the driver fromoperating the vehicle based on results of the prediction. An autonomousvehicle or a vehicle having advanced driver-assist features may increasethe level of automation or fully take over operation of the vehicledepending on side effects of medications and remaining half-life of themedications.

SUMMARY

An aspect is a driver monitor system for predicting impairment of a userof a vehicle, the system including at least one video camera; an inputoutput device for inputting a list of at least one medication beingtaken by the user of the vehicle; and processing circuitry configuredto: predict at least one side effect of the at least one medicationbased on the half-life of the at least one medication, detect eye gazemovement, eye lid position, and facial expression of the user usingimages from the at least one video camera, use the eye gaze movement,eye lid position, and facial expression to predict whether the user istransitioning into an impaired physical state that is a side effect ofthe at least one medication, verify the at least one side effect of theat least one medication, determine whether the user is fit to driveusing the verified at least one side effect of the at least onemedication, and output to the vehicle an instruction to operate thevehicle in a level of automation that makes up for the at least one sideeffect or to perform a safe pull over operation of the vehicle.

An aspect is a method of predicting impairment of a driver of a vehicleby a driver monitor system including at least one video camera, an inputoutput device for inputting a list of at least one medication beingtaken by the driver of the vehicle, and processing circuitry. The methodincluding predicting at least one side effect of the at least onemedication based on the half-life of the at least one medication,detecting, by the processing circuitry, eye gaze movement, eye lidposition, and facial expression using images from the at least one videocamera; using the eye gaze movement, eye lid position, and facialexpression to predict, by the processing circuitry, whether the user istransitioning into an impaired physical state; verifying the at leastone side effect of the at least one medication; determining whether theuser is fit to drive using the verified at least one side effect of theat least one medication; and outputting to the vehicle an instruction tooperate the vehicle in a level of automation that makes up for the atleast one side effect or to perform a safe pull over operation of thevehicle.

The foregoing general description of the illustrative embodiments andthe following detailed description thereof are merely exemplary aspectsof the teachings of this disclosure, and are not restrictive.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete appreciation of this disclosure and many of theattendant advantages thereof will be readily obtained as the samebecomes better understood by reference to the following detaileddescription when considered in connection with the accompanyingdrawings, wherein:

FIG. 1 is a diagram of levels of automated driving;

FIG. 2 illustrates a vehicle having an array of exterior sensors;

FIG. 3 is a block diagram of driver monitor system in accordance with anexemplary aspect of the disclosure;

FIG. 4 is a block diagram of an electronic control unit in accordancewith an exemplary aspect of the disclosure;

FIG. 5 is a system diagram of a medication monitoring app in accordancewith an exemplary aspect of the disclosure;

FIG. 6 is a block diagram of a computer system for a mobile displaydevice;

FIG. 7 is a schematic diagram of a human computer interface inaccordance with an exemplary aspect of the disclosure;

FIG. 8 is a flowchart for a method of predictive impairment monitoringin accordance with an exemplary aspect of the disclosure;

FIG. 9 is a medication label for an exemplary prescription drug;

FIG. 10 is a mediation label for an exemplary over-the-countermedication;

FIG. 11 is a display for a medication monitoring app for creating a listof medications in accordance with an exemplary aspect of the disclosure;

FIG. 12 is a display for a medication monitoring app for entry ofmedication status in accordance with an exemplary aspect of thedisclosure;

FIG. 13 is a flowchart of a method of evaluating medication side effectsin accordance with an exemplary aspect of the disclosure;

FIG. 14 is a display for a medication monitoring app for notifying adriver in accordance with an exemplary aspect of the disclosure;

FIG. 15 is a display for an in-vehicle navigation display device inaccordance with an exemplary aspect of the disclosure;

FIGS. 16A, 16B is a flowchart of the startup check of FIG. 8;

FIG. 17 is a flowchart of a method of driver monitoring in accordancewith an exemplary aspect of the disclosure;

FIGS. 18A, 18B are a block diagram of a reinforcement learning systemand an artificial neural network architecture in accordance with anexemplary aspect of the disclosure;

FIG. 19 is a flowchart of selecting a side effect for use in determininga level of automation in accordance with an exemplary aspect of thedisclosure; and

FIG. 20 is a flowchart of a method of determining a level of automaticdriving of FIG. 8.

DETAILED DESCRIPTION

In the drawings, like reference numerals designate identical orcorresponding parts throughout the several views. Further, as usedherein, the words “a,” “an” and the like generally carry a meaning of“one or more,” unless stated otherwise. The drawings are generally drawnto scale unless specified otherwise or illustrating schematic structuresor flowcharts.

Furthermore, the terms “approximately,” “approximate,” “about,” andsimilar terms generally refer to ranges that include the identifiedvalue within a margin of 20%, 10%, or preferably 5%, and any valuestherebetween.

Aspects of this disclosure are directed to a technique that evaluatesvarious medications, predicts side effects, and applies the mostappropriate response to the predicted side effect. The system utilizesvehicle sensors (e.g., cameras, blood pressure, heart rate, bodytemperature, etc.) and machine learning to predict side effects of takenmedications based on half-lives.

FIG. 2 illustrates a passenger vehicle having an array of exteriorsensors, such as those that may be found in passenger vehicles rangingfrom those equipped with advanced driver-assist features to thoseequipped as a fully autonomous vehicle, or self-driving vehicle.

Referring to FIG. 2, a vehicle 100 includes an array of sensors 103 anda controller, ECU 105. Sensors may be mounted on a roof of a vehicle,mounted on the vehicle body 101, and may be included within the body ofa passenger vehicle, or a combination thereof. The types of sensors thatmay be mounted on an exterior of a vehicle may include radar, LiDAR,video cameras, and sonar antennas. Video cameras, radar antennas, andsonar antennas may be located around a periphery of the vehicle. Inparticular, the passenger vehicle may be fitted with forward-lookingcameras to detect traffic signals, as well as front-mounted sensors todetect other vehicles, pedestrians, and obstacles, or to determinetraffic conditions, such as intersections and merging traffic lanes, inthe vehicle's vicinity. The combination of sensors may be used to assistdrivers in choosing the safest routes possible, or may provideinformation needed for operation of an autonomous vehicle. Inparticular, a passenger vehicle 100 may include other sensors foradvanced control and navigation, including GPS, odometry and internalmeasurement units.

A passenger vehicle 100 may further include sensors such one or morethermometers for monitoring the cabin environmental conditions atdifferent portions of the interior. The cabin of a vehicle may alsoinclude video cameras and infrared thermometer sensors for monitoringpersons and other objects within the vehicle cabin. A passenger vehiclemay include internal sensors for monitoring various conditions of thevehicle, such as steering angle and vehicle speed. Also, the vehicleengine may include various sensors for pressure, temperature, air flowand engine speed. Tires may include pressure sensors for measuring thetire pressure. Provided readings from some of the sensors, otherparameters may be estimated or measured, which are referred to asestimators. For example, fuel usage rate may be estimated based on milesdriven and change in fuel level reading. Also, temperature in the cabinmay be measured as a heat map that is determined by several infraredthermometers positioned throughout the cabin.

FIG. 3 is a block diagram of driver monitor system in accordance with anexemplary aspect of the disclosure. Driver monitor systems may beincluded in various types of vehicles to enhance driving safety.Passenger vehicles may be driven by drivers that may not have hadsufficient sleep or are driving for long periods of time. In a similarmanner, trucks may be driven by truck drivers for extended periods oftime. Truck driver safety is a concern when the truck driver does notget sufficient rest. Delivery trucks may be faced with driving in heavytraffic conditions and into neighborhoods or congested areas requiringutmost alertness. Thus, driver monitor systems include sensors, such asin-vehicle cameras, to monitor a driver's face, a driver's headposition, track the driver's eye movement, the driver's posture in aseat, even other physical state conditions such as heart rate and facialblood flow. The driver monitor systems may include sensors to monitorthe vehicle state, such as motion of the steering wheel and position ofthe vehicle relative to the road. To avoid driver distraction, thelighting for in-vehicle cameras may be infrared lighting.

Although a driver monitor system may be considered most beneficial forlow levels of automated driving where the driver is required to controlmost driving functions, higher levels of automated driving also requirealertness of the driver particularly in situations where the driver musttake over a driving control function possibly due to an emergencysituation. For example, a Level 3 vehicle may encounter emergencysituations on a highway that are beyond the capabilities of theautomated driving.

Regarding FIG. 3, the driver monitor system 300 may include one or morein-cabin cameras 311 and associated light sources 313. The drivermonitor cameras 311 and light sources 313 may be located at variouspositions in the cabin interior. The driver monitor cameras 311 maycapture video images for different functions. At least two drivermonitor cameras 311 may capture images of the driver's face and/or head.At least one driver monitor camera 311 may capture images of thedriver's body posture while seated. At least one driver monitor camera311 may be part of an eye tracking system.

The driver monitor system 300 may include other in-cabin sensors 315 fordetecting the state of the driver or condition of the cabin environment,such as one or more touch free thermometers. The driver monitor system300 may include a heart rate monitor 317. The heart rate monitor 317 maybe a device worn by a driver, such as a smart watch that includes aheart rate monitoring function. The heart rate monitor 317 may includesensors built into the vehicle, such as heart rate sensors positionedaround the perimeter of a steering wheel. The heart rate monitor 317 maybe a camera that monitors blood flow to the face. The heart rate monitor317 may include an operation to store a heart rate profile in a memoryin order to perform further analysis of the heart rate such as heartrate variability analysis.

The driver monitor system 300 may include at least one electroniccontrol unit (ECU) 105. The electronic control unit 105 may performvarious functions using data received from the sensors 311, 315, 317. Animage synthesis function 323 may combine images received from one ormore driver monitor cameras 311. The image synthesis function 323 maycombine images to form a single synthesized image without overlap.Alternatively, the image synthesis function 323 may combine two or moreimages to form a three dimensional image. The three dimensional imagemay be of a driver's face, or of a driver's head.

A facial information detection function 325 may use one or more imagesreceived from driver monitor cameras 311 and detect features of thedriver's face including eyes, nose, mouth, and possibly ears. Detectedfacial features may include the position of the eyes, nose and mouth,and whether both ears are visible. Detected features may include whetherthe driver's mouth is open, or that the driver is yawning. The facialinformation detection function 325 may determine the position and/ormovement of the driver's head.

The heartrate monitor 317 is a sensor that provides a signal thatrepresents a driver's heart rate. A heart rate monitor may use opticaltechnology, which sends light into the person's skin and reads the lightcoming back in to track pulse. A pulse oximeter detects pulse byilluminating the skin with light from a light-emitting diode and thenmeasuring the amount of light reflected to a photodiode as aphotoplethysmographic (PPG) signal. Other heart rate monitors measureheart rate with a transmitter that detects electrical activity.

A heart rate variability function 327 may receive the heart rate signaland perform an analysis to determine variability of the heart ratesignal. Hear rate variability (HRV) is the physiological phenomenon ofvariation in the time interval between heartbeats. It is measured by thevariation in the beat-to-beat interval. HRV may be measured by obtaininga continuous heart rate signal, or by acquiring the PPG signal. Amongmethods of analyzing heart rate variability are time domain methods orfrequency-domain methods. Frequency domain methods assign bands offrequency and then count the number of intervals between normal beats(NN) that match each band. The bands are typically high frequency (HF)from 0.15 to 0.4 Hz, low frequency (LF) from 0.04 to 0.15 Hz, and thevery low frequency (VLF) from 0.0033 to 0.04 Hz. An eye trackingfunction 329 measures either the point of gaze (where one is looking) orthe motion of an eye relative to the head. An eye tracker is a devicefor measuring eye positions and eye movement. Video-based eye trackerstypically use the corneal reflection (the first Purkinje image) and thecenter of the pupil as features to track over time.

Provided medication information, including medication half-life, facialinformation from the facial information detection function 325, heartrate variability, gaze and eye movement, state of the eye lid, a driverbehavior prediction function 331 may be used to predict whether a driveris getting tired, may suffer from dizziness or fainting, may be unableto concentrate, or may be transitioning to any other reduced cognitivestate that may be a result of side effects from medications being taken.As will be discussed further below, driver behavior prediction may beimplemented using a machine learning technique.

It has been determined that heart rate variability (HRV) is lower than aresting state when a person is in a decreased cognitive state. Afatigued state will likely have a slower heartbeat compared tobaseline/norm. Another cognitive state may be characterized by a lowHRV.

The eye tracking function 329 may be configured to measure PERCLOS,which is a measure of the percent of eyelid closure. A PERCLOSEmeasurement may also be used to detect a fatigue state or anothercognitive state. Some cognitive states may have a low PERCLOS, while afatigued state will generally have a higher PERCLOS (drooping eyelids).The eye tracking function 329 may be implemented with a high definitionvideo camera 311.

The eye tracking function 329 may also be configured to perform variousother measurements including pupil dilation, saccade, and gazeeccentricity. Saccades serve as a mechanism for fixation, rapid eyemovement. When scanning immediate surroundings or reading, human eyesmake saccadic movements and stop several times, moving very quicklybetween each stop. Human eyes move around, locating interesting parts ofa scene and building up a mental, three-dimensional map corresponding tothe scene. Measurements of saccade may include saccade velocity,acceleration, and frequency.

The eye tracking function 329 may be configured to perform measurementsof eye gaze eccentricity, which is a deviation in the driver's gaze.Measurements may also include duration of eye gaze.

A driver monitor camera may also be configured to differentiate acognitive tunneling state from a fatigued state by its ability torecognize yawning and other facial traits associated with sleepiness,such as eyebrow furrowing.

Lateral movement of a steering wheel may also be used to detect driverfatigue state and other reduced cognitive states. Measurements ofsteering wheel movement may include steering entropy; steering wheelvelocity and acceleration, and steering wheel reversal frequency. When adriver is in a tunneling state, the steering wheel may be unusuallyfixated. When a driver is in a fatigue state, movements of the steeringwheel may be unusually larger than normal.

A human machine interface (HMI) 341 may include devices for visual andaudio outputs as well as computer processing circuitry for navigationand infotainment.

A controller area network (CAN) 351 is a network that allows controllersand devices in a vehicle to communicate with each other without a hostcomputer. Among other things, a CAN 351 may provide information aboutthe performance of a vehicle, such as the wheel angle, vehicle speed andacceleration.

FIG. 4 is a block diagram of an electronic control unit in accordancewith an exemplary aspect of the disclosure. The electronic control unit105 may be based on a microcontroller. A microcontroller includesprocessing circuitry that may contain one or more processor cores (CPUs)along with memory (volatile and non-volatile) and programmableinput/output peripherals. Program memory in the form of flash, ROM,EPROM, or EEPROM is typically included on chip, as well as a secondaryRAM for data storage. In one embodiment, the electronic control unit 105is an integrated circuit board with a microcontroller 410. The boardincludes digital I/O pins 315, analog inputs 417, hardware serial ports413, a USB connection 411, a power jack 419, and a reset button 421.Other microcontroller configurations are possible. Variations caninclude the number of pins, whether or not the board includescommunication ports or a reset button.

In an exemplary embodiment, the microcontroller may be a RISC-basedmicrocontroller having flash memory 403, SRAM 407, EEPROM 405, generalpurpose I/O lines, general purpose registers, a real time counter, sixflexible timer/counters, an A/D converter 409, and a JTAG interface foron-chip debugging. It should be understood that other microcontrollersmay be used. Microcontrollers vary based on the number of processingcores, size of non-volatile memory, the size of data memory, as well aswhether or not it includes an A/D converter or D/A converter.

FIG. 5 is a system diagram of a medication monitoring app in accordancewith an exemplary aspect of the disclosure. In some embodiments, amedication monitoring mobile application 513, or app, may be installedin a mobile display device 550, such as a smartphone, tablet, or otherwireless device having a display or audible function, or both. Themobile application 513 may be in communication with the HMI 341 of thevehicle 100. The mobile application 513 may utilize cloud services 521,including access to a database 523. In some embodiments, the vehicle 100may include an inter-vehicle communication function to communicate withother vehicles, such as to exchange traffic conditions.

Upon installation of a mobile application 513, the mobile display device550 may be sent a message that indicates that an account has been set upfor use of the mobile application 513. The mobile display device 550 maydisplay an indication that the mobile application 513 has beeninstalled.

FIG. 6 is a block diagram of a display processing system for the mobiledisplay device in accordance with an exemplary aspect of the disclosure.In particular, FIG. 6 is a block diagram of a mobile display device 550.The display processing system 666 provides support for simultaneouscamera sensor inputs, video decoding and playback, location services,wireless communications, and cellular services. The display processingsystem 601 includes a central processing unit (CPU) 615, and may includea graphics processing unit (GPU) 611and a digital signal processor (DSP)613. The CPU 615 may include a memory, which may be any of several typesof volatile memory 607, including RAM, SDRAM, DDR SDRAM, to name a few.The DSP 613 may include one or more dedicated caches 603 in order toperform computer vision functions as well as machine learning functions.The GPU 611 performs graphics processing for a 4K resolution displaydevice. The GPU 611, DSP 613, CPU 615, Cache 603, and in someembodiments, a cellular modem 621, may all be contained in a singlesystem-on-chip (SOC) 601. The display processing system 666 may alsoinclude video processing circuitry 623 for video decoding and playback,location service circuitry, including GPS and dead reckoning, andconnectivity service circuitry, including WiFi and Bluetooth. Thedisplay processing system 666 may include one or more input/outputports, including USB connector(s), such as connectors for USB 2, USB 3,etc.

In some embodiments, the mobile application 513 may synchronize (sync)with a navigation system, infotainment system, of the in-vehicle humanmachine interface system 341, and may communicate instructions/commandsto the vehicle via the in-vehicle human machine interface system 341.FIG. 7 is a schematic diagram of a human machine interface in accordancewith an exemplary aspect of the disclosure. While the human machineinterface 341 is depicted in abstract with other vehicular components,the human machine interface 341 may be integrated with other systemcomponents of the vehicle 100 (see FIG. 2).

As shown in the example of FIG. 7, a vehicle navigation device 702communicates through audio/visual control unit 708, which communicateswith a sensor control unit 714 over a communication path 713 via vehiclenetwork cloud 712.

As may be appreciated, the communication path 713 of the vehicle network712 may be formed of a medium suitable for transmitting a signal suchas, for example, conductive wires, conductive traces, opticalwaveguides, or the like. Moreover, the communication path 713 can beformed from a combination of mediums capable of transmitting signals. Inone embodiment, the communication path 713 can comprise a combination ofconductive traces, conductive wires, connectors, and buses thatcooperate to permit the transmission of electrical data signals tocomponents such as processors, memories, sensors, input devices, outputdevices, and communication devices. Accordingly, the communication path713 may be provided by a vehicle bus, or combinations thereof, such asfor example, a Body Electronic Area Network (BEAN), a Controller AreaNetwork (CAN) bus configuration, an Audio Visual Communication-LocalArea Network (AVC-LAN) configuration, a Local Interconnect Network (LIN)configuration, a Vehicle Area Network (VAN) bus, and/or othercombinations of additional communication-system architectures to providecommunications between devices and systems of the vehicle.

The term “signal” relates to a waveform (e.g., electrical, optical,magnetic, mechanical or electromagnetic), such as DC, AC,sinusoidal-wave, triangular-wave, square-wave, vibration, and the like,capable of traveling through at least some of the mediums describedherein.

The sensor control unit 714 receives sensor data 716 from the audiblesensor device 721, sensory input device 723, and video sensor device725. For further example, the sensor data 716 operates to permit objectdetection external to the vehicle, such as other vehicles (includingvehicles occupying a parking location), roadway obstacles, trafficsignals, signs, trees, etc. The sensor data 716 allows the vehicle 100(see FIG. 2) to assess its environment in order to maximize safety forvehicle passengers and objects and/or people in the environment. Thesensor data 716 also provides information relating to a moving target,and to provide moving target indication (MTI) data.

As an example, the sensory input device 723 provides tactile orrelational changes in the ambient conditions of the vehicle, such as anapproaching person, object, vehicle, etc. The one or more of the sensoryinput devices 204 can be configured to capture changes in velocity,acceleration, and/or distance to objects relative to the travel of thevehicle 100, as well as an angle of approach. The sensory input devices733 may be provided by a Light Detection and Ranging (LIDAR) systemand/or milliwave radar devices. As an example, the sensory input devices733 may identify objects in the roadway (such as other vehicle, debris,etc.), and may identify moving objects adjacent the roadway that maypresent a hazard to the vehicle 100 (such as animals and/or debriscoming within the roadway).

Sensor data 716 relating to the video sensor devices 311 (see FIG. 3)operate to capture still-frame of and/or video images within associatedfields of view for display to the touch screen 706 of the vehiclenavigation device 702.

The audio/visual control unit 708 receives the sensor data 716 via thecommunication path 713 and vehicle network 712, and produces displaydata 709 for display by the touch screen 706. The audio/visual controlunit 708 also receives user input data 711 from the vehicle navigationdevice 702, which may be from the tactile input 704, microphone 750,eye-tracking input device 740, etc.

The audio/visual control unit 808 may include an antenna 720 forwireless communications 723 with user devices, such as a mobile device550.

The mobile device 550, by way of example, may be a device includinghardware (for example, chipsets, processors, memory, etc.) forcommunicatively coupling with a network cloud and/or directly with theaudio/visual control unit 708 via the antenna 720, and also includes anantenna for such wireless communication.

The antenna 720 may include one or more conductive elements thatinteract with electromagnetic signals transmitted by global positioningsystem satellites. The received signals may be transformed into a datasignal indicative of the location (for example, latitude and longitudepositions), and further indicative of the positioning of the device 550with respect a vehicle position, that can be indicated on a mapdisplayed via the touch screen 706, or otherwise displayed via thevehicle GUI 703.

The wireless communications 723 may be based on one or many wirelesscommunication system specifications. For example, wireless communicationsystems may operate in accordance with one or more standardsspecifications including, but not limited to, 3GPP (3rd GenerationPartnership Project), 4GPP (4th Generation Partnership Project), 5GPP(5th Generation Partnership Project), LTE (long term evolution), LTEAdvanced, RFID, IEEE 802.11, Bluetooth, Bluetooth low energy, AMPS(advanced mobile phone services), digital AMPS, GSM (global system formobile communications), CDMA (code division multiple access), LMDS(local multi-point distribution systems), MMDS(multi-channel-multi-point distribution systems), IrDA, Wireless USB,Z-Wave, ZigBee, and/or variations thereof.

The vehicle navigation device 702 includes, for example, tactile input704, a touch screen 706, microphone 750, and eye-tracking input device740. The touch screen 706 operates to provide visual output or graphicuser interfaces such as, for example, maps, navigation, entertainment,information, infotainment, and/or combinations thereof.

The touch screen 706 may include mediums capable of transmitting anoptical and/or visual output such as, for example, a cathode ray tube,light emitting diodes, a liquid crystal display, a plasma display, etc.Moreover, the touch screen 706 may, in addition to providing visualinformation, detect the presence and location of a tactile input upon asurface of or adjacent to the display. Accordingly, the display mayreceive mechanical input directly upon the visual output provided by thetouch screen 706. Additionally, it is noted that the touch screen 706can include at least one or more processors and one or more memorymodules.

The vehicle navigation device 702 may also include tactile input and/orcontrol inputs such that the communication path 713 communicativelycouples the tactile input to other control units and/or modules of thevehicle 100 (see FIG. 2). The tactile input data may be provided bydevices capable of transforming mechanical, optical, or electricalsignals into a data signal capable of being transmitted via thecommunication path 713.

The tactile input 704 may include a number of movable objects that eachtransform physical motion into a data signal that can be transmittedover the communication path 713 such as, for example, a button, aswitch, a knob, etc.

The touch screen 706 and the tactile input 704 may be combined as asingle module, and may operate as an audio head unit or an infotainmentsystem of the vehicle 100. The touch screen 706 and the tactile input704 can be separate from one another and operate as a single module byexchanging signals.

Touch screen 706 may include a display screen, such as a liquid crystaldisplay (LCD), light emitting diode (LED), plasma display or other twodimensional or three dimensional display that displays graphics, text orvideo in either monochrome or color in response to display data 709.

A built-in eye-tracking input device 740 includes a near-infrared lighttransmitter that projects a pattern of tracking signals 744 onto theeyes of the user 746. The built-in eye-tracking input device 740 alsoincludes a camera operable to take high-frame-rate images via thetracking signals 744 of the user's eyes and the reflected patterns. Inthis manner, the built-in eye-tracking input device 740 operates todetermine a gaze point 742 of the touch screen 706. As with a physicaltouch of the touch screen 706, the gaze point 742 may be used as a userinput, which is provided as user input data 711 to the audio/visualcontrol unit 708.

FIG. 8 is a flowchart for a method of predictive impairment monitoringin accordance with an exemplary aspect of the disclosure. Regarding FIG.8, a mobile app 513 or vehicle driver monitor system 300 may perform acheck as to whether the driver is fit to drive a vehicle before thevehicle is started. Medications that a driver is taking may be obtained,in S801, from a doctor prescription or an electronic medical record.Other medications that are taken occasionally may be entered, in S803,through the mobile app 513 or vehicle driver monitor system 300. Forexample, a driver may take an occasional medication such as melatonin tohelp sleep or an antihistamine for minor allergies. Although takingoccasional medications like sleep medications and antihistamines haverelatively short half-lives on the order of about one hour and may havefewer side effects, a driver may alter a dosage or take a medicationthat may cause side effects such as dizziness or daytime sleepiness. Forexample, taking a dosage of melatonin that is greater than the properdose of 1 to 5 mg can disrupt a person's circadian rhythm and causedaytime sleepiness. The occasional medications may be entered bycapturing an image of the medication label with the camera of thesmartphone using the mobile app 513.

FIG. 9 is a medication label for an exemplary prescription drug. Bycapturing an image of a label of a prescription drug, information aboutthe medication such as the drug name and strength, when and how oftenthe drug is to be taken may be obtained. The exemplary medication labelmay include a name and address of the pharmacy 901, the doctor's name903, a drugstore phone number 905, a prescription file date 907, anidentifier used by the drugstore for refills, the patient name 911,instructions about how often and when to take the drug 913, name of drugand strength of drug 915, number of refills 917, and a use-by date 919.

FIG. 10 is a mediation label for an exemplary over-the-countermedication. Similar to a prescription drug, the name and strength of themedication may be obtained by capturing an image of the medicationlabel. A label for an over-the-counter medication may vary depending onthe drug manufacturer. FIG. 10 is an exemplary medication label that maybe captured by a camera in the case of a manufacturer's label. Amanufacturer's label may include the brand name 1003, a drug name 1005,the strength of the drug 1007, the contents 1009 of the package. Thepackage may be identified by a universal product code number and anassociated barcode 1001. The universal product code includes themanufacturer's identification number and the product number. Theuniversal product code may be used to obtain further information aboutthe product from a database.

In some embodiments, a list of medications that are being taken may beentered manually or extracted from an image of a medication label tosupplement medications that are obtained from medical records. FIG. 11is a display for a medication monitoring app for creating a list ofmedications in accordance with an exemplary aspect of the disclosure. Ina mobile application 513 for monitoring medications with respect to adriver of a vehicle, the mobile application 513 may be provided with auser interface screen for adding or deleting from a list of one or moremedications 1111 being taken by the patient/driver. It is preferred thatthe list include the strength 513 of each medication. The mobileapplication 513 may include a form for manual entry of medications,where the added medications may be entered by using a camera function ofthe mobile device 550. The camera of the mobile device 550 may be usedto capture an image of a product label on the container of a medication.The mobile application 513 may perform optical character recognition(OCR) to extract information of the medication from the image of theproduct label. The mobile application 513 may extract a product ID,which may be used to obtain the name of the medication and dosage amountfrom a database, such as a database 523 maintained in the cloudservices.

The database 523 may be a relational database or a table in a flat filedepending on the size of the database. A table in a flat file may beused when the database is for an individual patient driver, whereas arelational database or some other database system may be used for adatabase of general medication information. The medication list of anindividual patient itself may be maintained in a table. Such a table maybe stored locally in the mobile device 550.

A database 523 of general medication information may include an ID thatis unique to a product in a particular container (e.g., a drugstore,manufacturer ID), or may be an ID established by the database thatidentifies the product (for example, an ID that is unique to theproduct, including the product name, form—tablet, gel, etc.,quantity—100 tablets, etc., strength). Other information identifying themedication contained in the database 523 may include the product name,generic drug name, strength, prescription fill date or a purchase date,a product expiration date. The database 523 may contain additionalinformation about medications such as instructions about how often andwhen a drug is to be taken and the class of the medication such asantidepressants, benzodiazepines, sleep medications, etc.

The database 523 may contain a half-life of a medication. The half-lifeof a drug is a pharmacokinetic parameter that may be defined as the timeit takes for the concentration of the drug in the plasma or the totalamount in the body to be reduced by 50%. In other words, after onehalf-life, the concentration of the drug in the body will be half of thestarting dose. As an example, if 100 mg of a drug with a half-life of 60minutes is taken, then 60 minutes after administration, 50 mg remains,after 120 minutes after administration, 25 mg remains, and after 180minutes, etc.

Also, the drug half-life varies for each person. The drug half-life canvary based on several factors including a person's weight, gender, age,blood circulation, diet, fluid levels (excessive or dehydrated), historyof previous drug use, kidney function, liver function, obesity,pre-existing conditions (heart failure, gastrointestinal disorders),presence of drugs that compete for binding sites or interact,race/ethnicity, smoking, and other factors.

For purposes of this disclosure, the half-life of a medication that isstored in the database 523 is a generic half-life of the medication thatmay be adjusted based on a person's weight.

The displayed list of medications may include a function to add amedication, for example in the form of a button 1115, and a function todelete a medication, for example in the form of button 1117. A functionmay also be included to edit the list of medications, such as in thecase that a dosage amount of a medication is changed or in the case thatthe name of the medication has changed. As indicated above, a medicationmay be added to the list using the camera function of the mobile device550. When a medication is to be added using the mobile application 513,the mobile application 513 may check a database 523 of medications inthe cloud services 521 for similar medication products, and/ormedication products that should not be combined with the medication tobe added. The information retrieved from the database 523 may be used bythe mobile application 513 to determine if there may be othermedications in the list of medications that may be redundant medications(i.e., providing an increased dosage of the same or similar drug), or ifthere are medications in the list that should not be taken together withan added medication. The mobile application 513 may include a functionto generate a report of possible multiple medications and possible druginteraction issues, and display the report. In some embodiments, theredundant medications and drug interaction issues may be well knowninformation that is used to check the list of medications 1111.

FIG. 12 is a display for a medication monitoring app for entry ofmedication status in accordance with an exemplary aspect of thedisclosure. In some embodiments, a driver may pull up the mobileapplication 513 to display a screen for managing status of medicationstaken. The status display may allow for entry of a time, or time of day,1213 that a medication 1211 in the previously entered medication list1211 is taken. In some embodiments, the status of a medication may beupdated by a function that detects the time and date that is maintainedin the mobile device 550. The updating of the status may be by way ofclicking on listed medications as they are taken. In addition, thedriver may enter additional drug-related products 1115, especially thosethat contain caffeine or alcohol. In the example display screen, thedriver has added a stimulant that is being consumed. The driver may alsoenter any over-the-counter products that have not already been enteredin the original list of medications 1111. For example, the driver mayhave taken an over-the-counter allergy medication because they believethey are presently having allergy problems. The allergy medication maybe non-drowsy type, or may be a drowsy type medication. Other nighttimemedications, even if taken a night before driving, can be entered withtheir most recent time.

In S805, a preliminary impairment calculation may be performed in themobile app 513 or driver monitor system 300 which takes into account thespecific medication and an associated medication half-life. Thepreliminary impairment calculation may be based on the time of day,regularity, and dosage indicated on the medication prescription.

FIG. 13 is a flowchart of a method of evaluating medication side effectsin accordance with an exemplary aspect of the disclosure. Thepreliminary impairment calculation, S805, may be determined based on atime since a medication is taken and utilizes the medication half-life.In S1301, the medications that a driver is taking are retrieved from thedatabase 523 based on those input in the mobile application 513, such asusing the screen shown in FIG. 5 and any additional medications enteredwhen the medication status screen 1101, such as in FIG. 11, isdisplayed. Medication names may be the name of the drug and/or may be ageneric drug name. In some cases a medication name may just be a drugclass depending on what is known about the medication. A strength 513 ofmedication is associated with the medication name in the medication list1111 and is also retrieved from the database 523. In addition, thehalf-life of each medication is retrieved from the database 523 based onthe medication name.

When medication side effects are determined, the current time isobtained. The current time may be obtained from the mobile device 550 orfrom some other accurate time source. Also, the time that eachmedication is taken may be obtained based on input in the status screen1101, such as in FIG. 11. In S1303, the period of time since taking amedication is determined using the time that each medication is taken.In S1305, a decision is made as to whether the half-life of eachmedication is reached?

In some embodiments, the medication side effects are stored in thedatabase 523. The side effects of a medication are considered maximumwhen the medication is first taken, but are calculated as a reducedeffect based on the medication half-life. Side effects may be consideredas being safe levels after the half-life of a medication. For example,if a medication half-life is one hour, a side effect of drowsiness maybe considered to be sufficiently alleviated such that it would be safeto drive after one hour.

When the half-life of each medication in the list of medications hasbeen reached (YES in S1305), in S1307, the mobile application 513outputs that there are no side effects from medications. When thehalf-life of at least one medication in the list of medications has notbeen reached (NO in S1305), in S1309, side effects for the at least onemedications are retrieved from the database 523.

In some embodiments, the mobile application may display a result of thepreliminary impairment calculation. FIG. 14 is a display for amedication monitoring app for notifying a driver in accordance with anexemplary aspect of the disclosure. The mobile application 513 may usethe results of the preliminary impairment calculation to make aprediction as to whether it would be safe for a driver to drive avehicle at a current time. The mobile application 513 may make theprediction based on processing performed in the cloud services 521, ormay make the prediction in the mobile device 550 itself, or may even usecomputer resources of the vehicle 100. FIG. 14 is an example display1401 of a notification message that has been prepared as a result of aprediction. The notification message may list 1411 the medications thathave been used in the prediction and an indication 1413 of an impactthat the medication may have on driving a vehicle. The impact mayinclude a side effect(s) of taking the medication, whether or not itwould be safe to drive, or a message concerning the medication taken. Inthe example display 1401, the mobile application 513 recommends to stoptaking an energy drink as the energy drink may actually have an effectof further reducing cognitive concentration when taken together with theother medications. There may be other situations in which the mobileapplication 513 makes a prediction that it would be unsafe to drive avehicle based on the dosage amount and timing of taking a medication.The mobile application 513 may instruct the vehicle 100 to take action,such as switch to a higher level of driver assistance, prevent thevehicle from being started, take over control of certain vehicleoperations, depending on the predicted state of the driver and thedriver assist features that a vehicle is equipped with.

FIG. 15 is a display for an in-vehicle navigation display device inaccordance with an exemplary aspect of the disclosure. The mobile device550 may synchronize with a display device of the vehicle 100 and maycommunicate instructions/commands to an in-vehicle computer system. FIG.15 is an example display in a case where the mobile device 550 is insynchronization with the in-vehicle navigation display device 1502. Theinformation that is displayed may be substantially the same as theinformation displayed for the mobile device 550, as in FIG. 14 forexample.

In S807, a startup check is performed which may involve a dual checkthat includes both a physical check and a questionnaire.

FIGS. 16A, 16B is a flowchart of the startup check of FIG. 8. In S1630,the driver monitor system may be used to check for physical signs ofside effects. Physical signs of drowsiness

FIG. 17 is a flowchart for determining a driver state in accordance withan exemplary aspect of the disclosure. A driver monitor system may beinitially set up off line for a driver or drivers of a vehicle. Thedriver or drivers may enter into a user interface information about thedriver, which may include creating a driver behavior profile.

When a driver enters a vehicle and sits in the driver's seat, in S1701,the ECU 105 may turn on certain sensors, including heart rate monitorsensors 317, in-cabin cameras 311 and light sources 313. In someembodiments, initial sensors 311, 315, 317 and light sources 313 may beturned on when the engine is started. In some embodiments, all interiorcabin sensors may be turned on. In some embodiments, only a subset ofin-cabin cameras 311 and light sources 313 may be turned on.

In S1703, the initial in-cabin cameras 311 and light sources 313 areused to acquire facial information of a driver. The driver's face maynot be facing forward or may not be facing toward the initial in-cabincameras 311. Additional or alternative in-cabin cameras 311 may beturned on in order to obtain an image of the driver's face. In additionto obtaining an image of a driver's face, particular features may bemonitored such as facial blood flow, head position, body position, andyawning.

In S1705, at least one in-cabin camera 311 may be used for eye trackingof the driver. As mentioned above, eye tracking may include functions ofmonitoring eye movement and eye gaze direction, and in particular,saccade velocity, acceleration, and frequency, and duration of eye gaze.

In S1707, heart rate sensors may be used to provide the heart rate ofthe driver and in turn the heart rate information may be used todetermine a heart rate variability pattern.

In S1711, the ECU 105 may detect a head position that may indicatesleepiness or may detect facial features that indicate yawning, anxietyor lack of concentration.

In S1715, the ECU 105 may detect that the driver's eyes have moved awayfrom the forward direction to a left, right, or downward eye gazedirection for a predetermined period of time.

In S1717, the ECU 105 may detect that the driver's heart ratevariability has changed by a predetermined amount. In some embodiments,the ECU 105 may detect that the driver's heart rate variability is belowthe resting heart rate variability for a predetermined period of time.The heart rate variability may be measured over a predetermined periodof time, such as in a range of two to five minutes.

In S1719, provided results of atypical facial features, eye movement,and the heart rate variability, as well as head movement and position,body position, the ECU 105 may classify a driver's physical and mentalstate. In preferred embodiments, the driver's physical and mental stateis normal, or one or more of conditions that are possible side effectsof medications.

In S1631 of FIG. 16A, a decision is made as to whether the driver'sphysical and mental state is a side effect of a medication.

Machine learning may be used to predict whether a driver is moving intoa side effect of a medication, such as a fatigue state, some otherreduced cognitive state, or other side effect. The machine learningmodel may be made off line using a supervised learning algorithm, suchas a Support Vector Machine (SVM) or regression analysis, or may be madeby a continuous learning algorithm, such as reinforcement learning.

FIG. 18A is a block diagram of a reinforcement learning system inaccordance with an exemplary aspect of the disclosure. FIG. 18B is anarchitecture for the artificial neural network of FIG. 18A.

In reinforcement learning, an agent 1810 interacts with an environment1820 in discrete time steps. Learning is performed in an artificialneural network 1813. The artificial neural network 1813 may be amulti-layered network having at least one hidden layer. The input layerof the network 1813 is arranged according to a vector representation ofthe state 1811. The output layer of the network 1813 will consist of oneneuron for each possible action. At each time t, the agent 1810 receivesan observation which typically includes the reward . It then chooses1815 an action from the set of available actions (output from the neuralnetwork 1813), which is subsequently sent to the environment 1820. Theenvironment 1820 moves to a new state and the reward associated with thetransition is determined. The goal of a reinforcement learning agent isto collect as much reward as possible. The agent 1810 can (possiblyrandomly) choose any action as a function of the history.

The driver monitor system 300 may include feedback input from the driverto train a machine learning algorithm. Reinforcement learning allows forcontinuous learning and may learn based on the driver feedback. Thedriver monitor system's 300 sensors (Observed state 1821) are fed to theartificial neural network 1813 which may detect a state of a driver. Anaction selector 1815 will select an action 1825, such as ask the driver,“Are you thinking about something intently right now?” for an out offocus state or “Are you feeling sleepy at this moment?” for fatigue. Apositive reward 1823 (e.g., +1) will be awarded when the answer to thequestion is Yes. A negative reward 1823 (e.g., −1) may be awarded whenthe answer to the question is No, or Not at all. A lower positive reward(e.g., +0.5) may be awarded when the answer to the question is Somewhat.The driver monitor system 300 may perform preprocessing 1817 of sensordata, including quantifying the sensor data. For example, a 3-pt scalemay be implemented (1—not at all, 2—somewhat, 3—yes) to help ordinatethe sensor data. This data 1811 is then fed back into the artificialneural network 1813 so that the system is able to more effectively andrapidly detect driver states for that specific driver and issue anappropriate action.

FIG. 18B is a block diagram of the architecture for the artificialneural network in accordance with an exemplary aspect of the disclosure.The architecture of the artificial neural network 1813 may include anencoder 1851, and a sequence of N tandem decoders 1853, 1855. Theencoder 1851 is a shallow artificial neural network with one hiddenlayer 1861. The encoder 1851 generates vector representations of theinput driver state. In one embodiment, after training the encoder 1851,the hidden layer 1861 represents the vector representation for thedriver state. Each decoder 1853, 1855 is a multilayer artificial neuralnetwork, each having at least two hidden layers 1863, 1865, 1867, i.e.,is a deep learning neural network. Although FIG. 18B shows threeartificial neural networks arranged as tandem decoders, the number ofdecoders may be different depending on desired accuracy and trainingtime. A decoder 1853 takes the vector representations as inputs andoutputs the driver class. During training, a subsequent decoder 1855takes the vector representations as inputs and uses the outputs of theupstream decoder 1853 as targets. This method of training is performedin other downstream decoders in sequence. Each decoder 1853, 1855may bean artificial neural network architecture that is trained using thebackpropagation algorithm. An output layer 1870 of the last decoderoutputs the actions based on a detected driver class.

The artificial neural network is trained by adjusting weightedconnections between the layers. These weighted connections, orparameters, of the artificial neural network may be stored in theprofile associated with a driver. In some embodiments, different driversmay have their own artificial neural network with associated parameters,which may be stored in independent profiles for each driver.

Each driver profile may include a resting HRV. A low HRV may be definedas an HRV that is below the resting HRV. A high HRV may be defined as anHRV that is above the resting HRV. This driver profile can be stored ina cloud-based database 523 and accessed by any vehicle with themedication monitoring mobile application 513.

As an alternative to reinforcement learning, a machine learning modelmay be determined using the driver profile and Support Vector Machines(SVM). A SVM is a binary classifier, meaning it classifies data samplesinto one of two classes. In the case of classifying a driver of being ina cognitive tunneling state, fatigue state, or some other state, theclassification requires classifying the data into three or more classes,a problem referred to as multiclass classification. One strategy tosolving the multiclass classification problem using binary classifiersis to train a single classifier for each class, known as one-vs-reststrategy. In the one-vs-rest strategy, each classifier produces areal-valued confidence score for its classification decision. Theclassifier having the highest confidence score is the likely class forthe data. The SVM may include a system of binary classifiers, where oneclassifier determines if the driver is in a cognitive tunneling state,or another state, a second classifier determines if the driver is in afatigue state, or another state, and a third classifier determines ifthe driver is in another state. The classifier having the highestconfidence score represents the likely driver state.

Sensor data for a driver state 1811 may include: medication status basedon medication half-life, head motion—head droop, head falling down, Yaw,Pitch; facial features including yawning; body posture in seat; heartrate variability (high, low, resting); facial blood flow (high, normal);eye gaze direction (straight ahead, left, right, down), eye movement(steady, moving), and PERCLOS (percentage).

In S1631, a determination is made as to whether physical and mentalstate of the driver as determined using sensors is within a tolerancerange. If one or more physical and mental state is within a tolerance ofside effects determined by the machine learning model (S1631, Pass), aquestionnaire of common side effects is provided to the driver. If thestate is not within a tolerance of side effects determined by themachine learning model (S1631, Fail), in S1633, the driver may request aquestionnaire in order to verify the results of the decision step S1631.The questionnaire, S1635, asks the driver a series of questions.

In some embodiments, the mobile application 513 or vehicle navigationdevice 702 will provide an inquiry in the form of a displayed message oran audible statement. The inquiry may be a question such as “Are youSleepy?” The mobile application 513 or vehicle navigation device 702 mayprovide an inquiry such as “is your vision blurry?” The mobileapplication 513 or vehicle navigation device 702 may provide an inquirysuch as “are you feeling dizzy or light-headed?” The mobile application513 or vehicle navigation device 702 may provide an inquiry such as “areyou finding your movement difficult?” The mobile application 513 orvehicle navigation device 702 may provide an inquiry such as “do youfind it difficult to focus or concentrate?”

The mobile application 513 or vehicle navigation device 702 may providean inquiry, such as “do you have a stomach ache?” In some embodiments,the mobile application 513 or vehicle navigation device 702 may providea further inquiry, such as “can you drive with driver assist?

In S1637, a check is made as to whether an answer in the questionnaireindicates failure. If the questionnaire is failed (S1637, fail), inS1641, negative feedback will be provided to the machine learning model.If the questionnaire is passed (S1637, pass), in S1639, positivefeedback will be provided to the machine learning model.

In S809, a decision is made as to whether the driver is fit to drive. Afitness of a driver to drive may be based on the level of automateddriving of a vehicle. A vehicle having a high level of automation mayprovide functions that alleviate or augment the driver's capacity todrive, while lower levels of automation may require greater capacity ofa driver. Also, side effects such as drowsiness or dizziness may be sosevere such that a driver may not have sufficient capacity to driveirrespective of the level of automation of a vehicle in all but thehighest level of automated driving. If it is determined that the driveris currently not fit to drive (NO in S809), in S811, the vehicle willnot be started.

In one embodiment, side effects including drowsiness or sleepiness,blurred vision, dizziness, and fainting may be a high degree of driverimpairment. This high degree of driver impairment may require that thevehicle not be driven by the driver until the side effects aresufficiently alleviated. In such case, the driver may have to consideralternative forms of transportation, such as rides with another personas the driver, taking a taxi cab, riding in a shuttle bus or van, or aform of public transportation.

In one embodiment, after a period of time the driver may decide thatthey are feeling better and may wish to proceed to re-perform the check.FIG. 16B is a flowchart for re-performing the startup check, S807. InS1651, the driver may enter into the mobile application 513 or vehiclenavigation device 702, an indication that they are feeling better andwould like to proceed with the startup check. In S1653, the mobileapplication 513 or vehicle navigation device 702 may perform the startupcheck of FIG. 16A.

Otherwise, the vehicle may be started (YES in S809), but, in S813, thedriver will be continuously monitored. Driver monitoring is a dynamicprocess that continuously cycles through determining medication sideeffects, acquiring driver facial information, performing eye tracking,monitoring heart rate, and acquiring vehicle information, as necessary.In monitoring the driver, the mobile app 513 or the driver monitorsystem 300 will check (in S815) whether the driver shows signs of sideeffects. Side effects from medications may include various degrees oroutcomes. Side effects such as slowed movement, inability to focus orconcentrate, nausea may vary by amount of slowed movement, amount ofinability to focus or concentrate, degree of nausea. Such variations ofoutcomes of the side effect decision will be provided for training amachine learning model, in S819. In order to determine a degree oroutcome of a side effect, a query may be provided to the driver, forexample, by asking a question: “are you feeling OK?”

If the response to the question verifies that the side effect issignificant enough to warrant limiting the driver's capacity to drive(NO in S817), the mobile app 513 or the driver monitor system 300 maydetermine, in S821, the level of automated driving of the vehicle. Ifthe level of automated driving is too low (NO in S821), the vehicle mayperform a safe pull over operation S825. If the level of automateddriving of the vehicle is high (YES in S821), the vehicle may beswitched to autonomous driving mode, S823.

In some embodiments, the autonomous driving mode, S823, may be a levelof automated driving that depends on the side effects. Side effectsincluding slowed movement, reduced or inability to focus or concentrate,sever nausea, and some other side effects may be associated with a lowdegree of driver impairment. This low degree of driver impairment mayrequire some amount of driver assist or autonomous piloting in order tosafely drive the vehicle. The particular driver assist or autonomouspiloting functions may depend on the type or extent of a side effect. Ofcourse a driver may suffer from multiple side effects.

In some embodiments, when there are multiple side effects, the mobileapplication 513 or vehicle navigation device 702 may select a sideeffect. FIG. 19 is a flowchart of selecting a side effect for use indetermining a level of automation in accordance with an exemplary aspectof the disclosure.

Some side effects may be more severe than other side effects in theirpossible effect on a driver's ability to drive. The method in FIG. 19may result in different levels of automatic driving depending on theexpected side effects of medications. A medication may have more thanone side effect. Also, the driver may have taken more than onemedication, each having one or more side effects. In either case, thelevel of automatic driving will be based on the side effect that willcause the greatest degree of impairment on the ability of the driver todrive.

Regarding FIG. 19, in S1901, side effects may be sorted by expecteddegree of driver impairment. Some side effects may have the sameexpected degree of driver impairment, in which case all side effectshaving the same degree of driver impairment may be selected in S1903. InS1905, a level of automatic driving will be determined for vehiclecontrol in accordance with the selected side effects.

The particular driver assist or autonomous piloting functions may beselected to augment any deficiency that a driver may have as a result ofthe side effect. For example, in the case of slowed movement or slowreaction time, external vehicle sensors may monitor nearby vehicles orother objects so that vehicle actions such as breaking or steering maybe adjusted as necessary to avoid collision even if the driver is slowto respond. In some embodiments, vehicle functions such as breaking orsteering may be completely performed by the vehicle, whereas the driveris required to be attentive in case of emergency or where some automatedvehicle functions stop working.

FIG. 20 is a flowchart of a method of determining a level of automaticdriving of FIG. 8. The method as shown in FIG. 20 includes a series ofdecision steps for each type of side effect. It should be understoodthat other decision steps may be included for additional side effects.Decision steps having the same action step may be combined into a singledecision step. In some embodiments, two or more decision steps may beperformed in parallel. In some embodiments, decision steps may beproceeded by an inquiry to the driver.

The actions 1825 that may be selected by the agent 1810 may include astimulus to mitigate a driver state or may include activation ofadvanced driving safety features, or more strict vehicle actions such aspreventing the vehicle from starting or shutting down the vehicle beforeit is driven, depending on the expected ability of the driver to drivethe vehicle or take over driving in the case that an autonomous vehiclerequires some degree of manual control.

Regarding FIG. 20, in S2001, a check is made as to whether a side effectis drowsiness/sleepiness. In some embodiments, the mobile application513 or vehicle navigation device 702 will provide an inquiry in the formof a displayed message or an audible statement. In S2003, the inquirymay be a question such as “Are you Sleepy?” In S2005, the mobileapplication 513 or vehicle navigation device 702 will detect whether aresponse from the driver is positive (YES in S2005), or that the driverbelieves that he/she is not sleepy (NO in S2005). In the case that thedriver is drowsy/sleepy, in S2009, the mobile application 513 or vehiclenavigation device 702 will send an instruction/command to the vehicleECU 105 to perform a safe pull over of the vehicle.

In S2007, a check is made as to whether a side effect is blurred vision.Again, the mobile application 513 or vehicle navigation device 702 mayprovide an inquiry before taking action, then, in S2009, sending aninstruction/command to the vehicle ECU 105 to safe pull over of thevehicle.

In S2011, a check is made as to whether a side effect is dizziness.Again, the mobile application 513 or vehicle navigation device 702 mayprovide an inquiry before taking action, then, in S2013, sending aninstruction/command to the vehicle ECU 105 to perform safe pull over ofthe vehicle.

In S2015, a check is made as to whether a side effect is fainting.Again, the mobile application 513 or vehicle navigation device 702 mayprovide an inquiry before taking action such as, in S2017, sending aninstruction/command to the vehicle ECU 105 to shut down the vehicle orprevent the vehicle from being started.

In S2019, a check is made as to whether a side effect is slowedmovement. Again, the mobile application 513 or vehicle navigation device702 may provide an inquiry before taking action, then, in S2021, sendingan instruction/command to the vehicle ECU 105 to perform driver assistfunctions or autonomous piloting of the vehicle 100. Driver assistfunctions may include stepped up breaking when the vehicle 100 is withina certain distance from another vehicle being followed. Driver assistfunctions may include anticipating movement of the steering wheel basedon a foreseen curve in the road.

In S2023, a check is made as to whether a side effect is a reducedability to focus or concentrate. Again, the mobile application 513 orvehicle navigation device 702 may provide an inquiry before takingaction such as, in S2025, sending an instruction/command to the vehicleECU 105 to perform driver assist functions or autonomous piloting of thevehicle 100.

In S2027, a check is made as to whether a side effect is nausea. Themobile application 513 or vehicle navigation device 702 may provide aninquiry, such as S2029 of “do you have a stomach ache?” In someembodiments, the mobile application 513 or vehicle navigation device 702may provide a further inquiry, such as S2035 of “can you drive withdriver assist? If so (YES in S2031 and S2037), in S2025, the mobileapplication 513 or vehicle navigation device 702 may send aninstruction/command to the vehicle ECU 105 to perform driver assistfunctions or autonomous piloting of the vehicle 100.

In S2033, a check is made as to whether there is another side effect.Again, the mobile application 513 or vehicle navigation device 702 mayprovide an inquiry, such as S2035 of “can you drive with driver assist?”If so (YES in S2037), in S2025, the mobile application 513 or vehiclenavigation device 702 may send an instruction/command to the vehicle ECU105 to perform driver assist functions or autonomous piloting of thevehicle 100.

Numerous modifications and variations of the present invention arepossible in light of the above teachings. It is therefore to beunderstood that within the scope of the appended claims, the inventionmay be practiced otherwise than as specifically described herein.

1. A driver monitor system for predicting impairment of a user of avehicle, the system comprising: at least one video camera; an inputoutput device for inputting a list of at least one medication beingtaken by the user of the vehicle; and processing circuitry configuredto: predict at least one side effect of the at least one medicationbased on the half-life of the at least one medication, detect eye gazemovement, eye lid position, and facial expression of the user usingimages from the at least one video camera, use the eye gaze movement,eye lid position, and facial expression to predict whether the user istransitioning into an impaired physical state that is a side effect ofthe at least one medication, verify the at least one side effect of theat least one medication, determine whether the user is fit to driveusing the verified at least one side effect of the at least onemedication, and output to the vehicle an instruction to operate thevehicle in a level of automation that makes up for the at least one sideeffect or to perform a safe pull over operation of the vehicle.
 2. Thedriver monitor system of claim 1, wherein the input output device isconfigured to display a status list of the at least one medication and amost recent time that the at least one medication had been taken by theuser.
 3. The driver monitor system of claim 2, wherein the input outputdevice is configured to add a medication to the status list of the atleast one medication.
 4. The driver monitor system of claim 2, whereinthe processing circuitry is configured to predict at least one sideeffect of the at least one medication based on the half-life of themedication by determining if the half-life of the at least onemedication has been reached using the most recent time that the at leastone medication had been taken by the driver.
 5. The driver monitorsystem of claim 1, further including a machine learning device, whereinthe input output device outputs a verification request and receives aresponse to the verification request, and wherein the eye gaze movement,the eye lid position, and the facial expression are fed back to themachine learning device which learns to predict whether the driver istransitioning into an impaired physical state.
 6. The driver monitorsystem of claim 5, wherein parameters of the machine learning devicethat are learned are stored in a memory as a profile associated with theuser.
 7. The driver monitor system of claim 6, wherein independentprofiles are stored in the memory in association with respectivedifferent users.
 8. The driver monitor system of claim 5, wherein theprocessing circuitry further monitors eye gaze movement, and wherein theeye gaze movement is fed back to the machine learning device whichlearns to predict whether the driver is transitioning into an impairedphysical state.
 9. The driver monitor system of claim 5, wherein themachine learning device learns by performing a reinforcement learningalgorithm.
 10. The driver monitor system of claim 1, wherein theprocessing circuitry is configured to predict a side effect of the atleast one medication including sorting side effects by expected degreeof user impairment, and to select at least one side effect having ahighest degree of user impairment.
 11. A method of predicting impairmentof a driver of a vehicle by a driver monitor system including at leastone video camera, an input output device for inputting a list of atleast one medication being taken by the driver of the vehicle, andprocessing circuitry, the method comprising: predicting at least oneside effect of the at least one medication based on the half-life of theat least one medication, detecting, by the processing circuitry, eyegaze movement, eye lid position, and facial expression using images fromthe at least one video camera; using the eye gaze movement, eye lidposition, and facial expression to predict, by the processing circuitry,whether the user is transitioning into an impaired physical state;verifying the at least one side effect of the at least one medication;determining whether the user is fit to drive using the verified at leastone side effect of the at least one medication; and outputting to thevehicle an instruction to operate the vehicle in a level of automationthat makes up for the at least one side effect or to perform a safe pullover operation of the vehicle.
 12. The method of claim 11, furthercomprising: displaying, by the input output device, a status list of theat least one medication and a most recent time that the at least onemedication had been taken by the user.
 13. The method of claim 12,further comprising: adding, by the input output device, a medication tothe status list of the at least one medication.
 14. The method of claim12, further comprising: predicting, by the processing circuitry, atleast one side effect of the at least one medication based on thehalf-life of the medication by determining if the half-life of the atleast one medication has been reached using the most recent time thatthe at least one medication had been taken by the user.
 15. The methodof claim 11, the system further including a machine learning device, themethod further comprising: outputting, by the input output device, averification request and receiving a response to the verificationrequest; and feeding back the eye gaze movement, the eye lid position,and the facial expression to the machine learning device which learns topredict whether the driver is transitioning into an impaired physicalstate.
 16. The method of claim 15, further comprising: storing in amemory parameters of the machine learning device that are learned as aprofile associated with the user.
 17. The method of claim 16, furthercomprising: storing in the memory independent profiles in associationwith respective different users.
 18. The method of claim 15, furthercomprising: monitoring, by the processing circuitry, eye gaze movement,and feeding back the eye gaze movement to the machine learning devicewhich learns to predict whether the driver is transitioning into animpaired physical state.
 19. The method of claim 15, wherein the machinelearning device learns by performing a reinforcement learning algorithm.20. The method of claim 11, wherein the predicting a side effect of theat least one medication includes sorting side effects by expected degreeof user impairment, and selecting at least one side effect having ahighest degree of user impairment.